Skip to content

Fix must-gather hang: apply --timeout to log streaming phase#2297

Open
afcollins wants to merge 1 commit into
openshift:mainfrom
afcollins:prow-timeout
Open

Fix must-gather hang: apply --timeout to log streaming phase#2297
afcollins wants to merge 1 commit into
openshift:mainfrom
afcollins:prow-timeout

Conversation

@afcollins

@afcollins afcollins commented Jul 2, 2026

Copy link
Copy Markdown

getGatherContainerLogs had no timeout, blocking forever if gather scripts ran long. Wrap all gather phases in a shared deadline context.

Summary by CodeRabbit

  • Bug Fixes
    • Improved gather reliability by applying a single timeout across the gather lifecycle, including container startup, log streaming, and completion checks.
    • Reduced noisy errors when gather log streaming stops due to cancellation or timeout.
    • Ensured the gathered output download/copy step continues independently so it can finish after the gather process completes.

@openshift-ci openshift-ci Bot requested review from atiratree and ingvagabund July 2, 2026 23:13
@openshift-ci

openshift-ci Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: afcollins
Once this PR has been reviewed and has the lgtm label, please assign atiratree for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown

Walkthrough

The must-gather log flow now uses a timeout-scoped context for log streaming and gather completion polling, while the download/copy step remains on the parent context. Log-streaming errors now ignore both cancellation and deadline expiration.

Changes

Must-Gather Timeout Scoping

Layer / File(s) Summary
Timeout-scoped gather flow
pkg/cli/admin/mustgather/mustgather.go
getGatherContainerLogs creates and cancels a timeout-scoped gatherCtx for log streaming and isGatherDone polling, and processNextWorkItem now suppresses both context.Canceled and context.DeadlineExceeded from log-streaming failures.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed Only pkg/cli/admin/mustgather/mustgather.go changed, and it adds no Ginkgo test titles; no unstable test names were introduced.
Test Structure And Quality ✅ Passed This PR only changes pkg/cli/admin/mustgather/mustgather.go and does not add or modify any Ginkgo tests, so the test-quality checklist is not applicable.
Microshift Test Compatibility ✅ Passed Only pkg/cli/admin/mustgather/mustgather.go changed; no new Ginkgo tests or MicroShift-unsafe OpenShift API usage was added.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo/e2e tests were added; only pkg/cli/admin/mustgather/mustgather.go changed, so SNO compatibility review is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed The PR only changes must-gather timeout/log streaming behavior; it adds no scheduling constraints, manifests, or topology-dependent controller logic.
Ote Binary Stdout Contract ✅ Passed Changed code only adjusts gather timeout handling; it adds no stdout writes in main/init/TestMain/suite-setup paths.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR only changes pkg/cli/admin/mustgather/mustgather.go; no new Ginkgo e2e tests or test files were added, so this check isn't implicated.
No-Weak-Crypto ✅ Passed Touched code only adds timeout/context handling and log-streaming behavior; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons found.
Container-Privileges ✅ Passed Patch only changes gather timeout/logging behavior in mustgather.go; no manifest or container privilege settings were added or modified.
No-Sensitive-Data-In-Logs ✅ Passed The patch only adds timeout context handling and suppresses deadline log noise; it introduces no new logging of sensitive data.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: applying --timeout to the must-gather log streaming phase to prevent hangs.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
pkg/cli/admin/mustgather/mustgather.go (1)

834-854: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add/update unit test coverage for the new timeout scoping behavior.

This change alters lifecycle timing and error-suppression logic but no corresponding test changes are included. processNextWorkItem is exercised via MustGatherOptions, so a table-driven test simulating a slow/hanging log stream (context deadline expiring mid-stream) would validate that the command now returns promptly with "gather never finished" instead of hanging, and that DeadlineExceeded from log streaming doesn't spuriously log "gather logs unavailable."
As per coding guidelines, "All changes must include unit test additions/changes."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/cli/admin/mustgather/mustgather.go` around lines 834 - 854, Add or update
unit tests for the new timeout scoping in MustGatherOptions.processNextWorkItem.
Cover a slow or hanging getGatherContainerLogs path where gatherCtx expires
mid-stream, and assert the command returns promptly with the expected "gather
never finished" behavior instead of hanging. Also verify that
context.DeadlineExceeded or context.Canceled from getGatherContainerLogs does
not trigger the "gather logs unavailable" log path, while non-context errors
still do.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/cli/admin/mustgather/mustgather.go`:
- Around line 834-854: Add or update unit tests for the new timeout scoping in
MustGatherOptions.processNextWorkItem. Cover a slow or hanging
getGatherContainerLogs path where gatherCtx expires mid-stream, and assert the
command returns promptly with the expected "gather never finished" behavior
instead of hanging. Also verify that context.DeadlineExceeded or
context.Canceled from getGatherContainerLogs does not trigger the "gather logs
unavailable" log path, while non-context errors still do.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 79101a03-76c9-4e86-9a09-f4f771ba4057

📥 Commits

Reviewing files that changed from the base of the PR and between a7ad572 and 2e8f8a3.

📒 Files selected for processing (1)
  • pkg/cli/admin/mustgather/mustgather.go

Comment thread pkg/cli/admin/mustgather/mustgather.go Outdated

// wait for gather container to be running (gather is running)
if err := o.waitForGatherContainerRunning(ctx, pod); err != nil {
if err := o.waitForGatherContainerRunning(gatherCtx, pod); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use ctx here. No need to include this kind of waiting in the timeout IMO.

Comment thread pkg/cli/admin/mustgather/mustgather.go Outdated
// wait for pod to be running (gather has completed)
log("waiting for gather to complete")
if err := o.waitForGatherToComplete(ctx, pod); err != nil {
if err := o.waitForGatherToComplete(gatherCtx, pod); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you are passing a timeout context here, we don't need to use PollUntilContextTimeout and o.Timeout within this function, so it can be removed, I think. Not that it really matters, but better not to repeat the same thing on multiple places.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I think we could revert to ctx here as well.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although, the difference between passing gatherCtx and the PollUntil loop within each function is each one of those loops will timeout at 35 minutes. Instead of the overall gather timing out at the shared timeout.

Is that the behavior we want?

I think the user experience of setting --timeout is that I expect the command to terminate after that timeout is reached, not that the timeout is applied for sub steps of the overall command.

@tchap

tchap commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Also I am not sure about our policy, but I tend to remove Co-Authored-By: Claude Opus 4.6 from commits. In Kubernetes it's actually forbidden.

@afcollins

afcollins commented Jul 6, 2026

Copy link
Copy Markdown
Author

Also I am not sure about our policy, but I tend to remove Co-Authored-By: Claude Opus 4.6 from commits. In Kubernetes it's actually forbidden.

I thought Red Hat policy was to identify commits so we know which ones are AI-assisted.

Anyway, removed. Please check again.

getGatherContainerLogs had no timeout, blocking forever if gather
scripts ran long. Wrap all gather phases in a shared deadline context.

Review feedback

Revert context from two loops that already manage timeout on their own.
Move timeout context within narrow function scope.

Signed-off-by: Andrew Collins <ancollin@redhat.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
pkg/cli/admin/mustgather/mustgather.go (1)

953-991: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Per-phase timeout budgets can compound beyond the documented --timeout semantics.

getGatherContainerLogs now derives its own gatherCtx with a fresh o.Timeout budget, on top of waitForGatherContainerRunning (line 1058, unchanged) already independently applying its own o.Timeout via wait.PollUntilContextTimeout. If waitForGatherToComplete follows the same pattern, the overall "gather" phase in processNextWorkItem (container-start wait → log stream → completion wait) could take up to ~3× o.Timeout in the worst case, not the single bounded o.Timeout that the flag's own help text promises ("This timeout only applies to the data gathering phase"). This is the same concern previously raised by reviewers (tchap: "we don't need to use PollUntilContextTimeout and o.Timeout within this function"; afcollins: "I think the user experience of setting --timeout is that I expect the command to terminate after that timeout is reached"), and it does not appear resolved here.

Separately, the comment at lines 954-956 claims gatherCtx covers "the entire gather lifecycle (waiting for container start, streaming logs, waiting for completion)", but gatherCtx is scoped only to this function's log-streaming/isGatherDone loop — waitForGatherContainerRunning and waitForGatherToComplete (called before/after, in processNextWorkItem) still use the plain ctx with their own separate o.Timeout windows. The comment should be corrected to reflect the actual (per-substep) scope, or the design should be changed to derive one shared deadline-scoped context in processNextWorkItem and pass it through all three phases (dropping the redundant internal o.Timeout usage in each), consistent with the earlier reviewer feedback.

Suggested direction (illustrative, needs coordination across the three phases)
 func (o *MustGatherOptions) processNextWorkItem(ctx context.Context, ns string, pod *corev1.Pod) error {
+	// Single deadline covering the whole gather phase (start-wait, log
+	// streaming, completion-wait). Copying intentionally continues on ctx.
+	gatherCtx, gatherCancel := context.WithTimeout(ctx, o.Timeout)
+	defer gatherCancel()
 	...
-	if err := o.waitForGatherContainerRunning(ctx, pod); err != nil {
+	if err := o.waitForGatherContainerRunning(gatherCtx, pod); err != nil {
 	...
-	if err := o.getGatherContainerLogs(ctx, pod); err != nil {
+	if err := o.getGatherContainerLogs(gatherCtx, pod); err != nil {
 	...
-	if err := o.waitForGatherToComplete(ctx, pod); err != nil {
+	if err := o.waitForGatherToComplete(gatherCtx, pod); err != nil {

And drop the now-redundant internal context.WithTimeout(ctx, o.Timeout) / o.Timeout-based polling inside getGatherContainerLogs / waitForGatherContainerRunning / waitForGatherToComplete.

As per coding guidelines, pkg/cli/**/*.go should use context.Context correctly for cancellation and timeouts — the concern here is that the current per-function contexts don't yield the single-overall-deadline behavior the --timeout flag documents.

#!/bin/bash
# Confirm whether waitForGatherToComplete independently re-applies o.Timeout
ast-grep run --pattern 'func (o *MustGatherOptions) waitForGatherToComplete($_, $_) $_ {
  $$$
}' --lang go pkg/cli/admin/mustgather/mustgather.go
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/cli/admin/mustgather/mustgather.go` around lines 953 - 991, The
gather-phase timeout handling is still using separate per-step deadlines instead
of one shared `--timeout` budget, so `processNextWorkItem`,
`waitForGatherContainerRunning`, `getGatherContainerLogs`, and
`waitForGatherToComplete` can compound their waits. Refactor the flow to derive
a single deadline-scoped context once in `processNextWorkItem` and pass it
through all three phases, removing the inner `context.WithTimeout(ctx,
o.Timeout)` / `wait.PollUntilContextTimeout` usage. Also update the
`getGatherContainerLogs` comment to match the actual scope if you keep any
substep-specific context behavior.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@pkg/cli/admin/mustgather/mustgather.go`:
- Around line 953-991: The gather-phase timeout handling is still using separate
per-step deadlines instead of one shared `--timeout` budget, so
`processNextWorkItem`, `waitForGatherContainerRunning`,
`getGatherContainerLogs`, and `waitForGatherToComplete` can compound their
waits. Refactor the flow to derive a single deadline-scoped context once in
`processNextWorkItem` and pass it through all three phases, removing the inner
`context.WithTimeout(ctx, o.Timeout)` / `wait.PollUntilContextTimeout` usage.
Also update the `getGatherContainerLogs` comment to match the actual scope if
you keep any substep-specific context behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 765d1b13-c49c-456d-a2c0-5e88b8cae86e

📥 Commits

Reviewing files that changed from the base of the PR and between 2e8f8a3 and 8d14830.

📒 Files selected for processing (1)
  • pkg/cli/admin/mustgather/mustgather.go

@openshift-ci

openshift-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

@afcollins: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants